Skip to content

[vlm, data] feat: expose Energon read-order controls - #139

Merged
kaimo455 merged 1 commit into
masterfrom
feature/vlm-data-shuffling
Aug 3, 2026
Merged

[vlm, data] feat: expose Energon read-order controls#139
kaimo455 merged 1 commit into
masterfrom
feature/vlm-data-shuffling

Conversation

@kaimo455

@kaimo455 kaimo455 commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • expose --data-shuffle-buffer-size and --data-max-samples-per-sequence for multimodal training
  • forward the resolved controls to both direct and blended Energon dataset paths
  • report the active read-order settings on rank zero while preserving existing defaults
  • add regression coverage for value resolution, CLI parsing, and both loader branches

Motivation

Offline-packed WebDataset records can contain long runs of similarly sized packs. These controls let users increase sample-level randomization and create shorter shard sequences without changing the default data pipeline.

Validation

  • uv run --no-project --isolated --with pytest pytest -q tests/test_vlm_dataloader_read_order.py — 6 passed
  • python3 -m py_compile loongforge/data/multimodal/dataloader_provider.py loongforge/train/arguments.py tests/test_vlm_dataloader_read_order.py
  • python -m build --sdist --wheel
  • twine check on the generated source and wheel distributions
  • pre-commit run spdx-check on all changed Python files
  • git diff --check

@kaimo455 kaimo455 self-assigned this Aug 3, 2026
@kaimo455
kaimo455 marked this pull request as ready for review August 3, 2026 04:03
@kaimo455
kaimo455 merged commit aa6ace4 into master Aug 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant